Hybrid agent-oriented object model to provide software fault tolerance between distributed processor nodes

ABSTRACT

An apparatus and method for a computer system is used for implementing an extended distributed recovery block fault tolerance scheme. The computer system includes a supervisory node, an active node and a standby node. Each of the nodes has a primary routine, an alternate routine and an acceptance test for testing the output of the routines. Each node also includes a device driver, a monitor and a node manager for determining the operational configuration of the node. The supervisory node coordinates the operation of the active and standby nodes. The primary and alternate routines are implemented with an application task through a plurality of agent objects operating as finite state machines. A reliable data link extends between the monitors of the active and standby nodes.

CROSS REFERENCE TO RELATED APPLICATIONS

The present allocation is a divisional application of and claimspriority benefit of U.S. patent application Ser. No. 10/183,488,entitled “HYBRID AGENT-ORIENTED OBJECT MODEL TO PROVIDE SOFTWARE FAULTTOLERANCE BETWEEN DISTRIBUTED PROCESSOR NODES” filed Jun. 28, 2002.

The present application is related to co-pending and commonly assignedPCT International Application No. PCT/US02/03323 entitled “DynamicBandwidth Allocation”, PCT/US02/03322 entitled “Demodulator BurstyController Profile”, PCT/US02/03193 entitled “Demodulator StateController”, PCT/US02/03189 entitled “Frame to Frame TimingSynchronization”, the disclosures of which are hereby incorporatedherein by reference. The aforementioned applications are related tocommonly assigned U.S. Pat. No. 6,016,313 entitled “System and Methodfor Broadband Millimeter Wave Data Communication” issued Jan. 18, 2000and currently undergoing two re-examinations under application Ser. No.90/005,726 and application Ser. No. 90/005,974, U.S. Pat. No. 6,404,755entitled “Multi-Level Information Mapping System and Method” issued Jun.11, 2002, U.S. patent application Ser. No. 09/604,437, entitled“Maximizing Efficiency in a Multi-Carrier Time Division Duplex SystemEmploying Dynamic Asymmetry”, which are a continuation-in-part of theU.S. Pat. No. 6,016,313 which are hereby incorporated herein byreference.

The present application is related to and is being concurrently filedwith commonly assigned U.S. patent application Ser. No. 10/183,383,entitled “Look-Up Table for QRT”; U.S. patent application Ser. No.10/183,486, entitled “Airlink TDD Frame Format”; U.S. patent applicationSer. No. 10/183,492, entitled, “Data-Driven Interface Control Circuitand Network Performance Monitoring System and Method”; U.S. patentapplication Ser. No. 10/183,490, entitled “Virtual Sector Provisioningand Network Configuration System and Method”, U.S. patent applicationSer. No. 10/183,489, entitled “System and Method for SupportingAutomatic Protection Switching Between Multiple Node Pairs Using CommonAgent Architecture”, U.S. patent application Ser. No. 10/183,384,entitled “System and Method for Transmitting Highly Correlated Preamblesin QAM Constellations”, the disclosures of which is hereby incorporatedherein by reference.

BACKGROUND OF THE INVENTION

A distributed recovery block is a method of integrating hardware andsoftware fault tolerance in a single structure without having to resortto N-version programming. In N-version programming, the goal is todesign and code the software module n times and vote on the n resultsproduced by these modules. The recovery block structure represents adynamic redundancy approach to software fault tolerance. In dynamicredundancy, a single program or module is executed and the result issubject to an acceptance test. Alternate versions are invoked only ifthe acceptance test fails. The selection of the routine is made duringprogram execution. In its simplest form as shown in FIG. 1, a standardrecovery block structure 100 consists of: a primary routine 110, whichexecutes critical software function; an acceptance test 120, which testthe output of the primary routine after each execution; at lease onealternate routine 115 that performs the same function as the primaryroutine and is invoked by the acceptance test 120 upon detection of afailure.

In a distributed recovery block 101 the primary and alternate routinesare both replicated and are resident on two or more nodes interconnectedby a network. This technique enables standby sparing fault tolerancewhere one node 105 a (the active node) is designated primary and anothernode 105 b (the standby node) is a backup. Under fault-freecircumstances, the primary node 105 a runs the primary routine 110whereas the backup node 105 b runs the alternate routine 115concurrently.

In case of a failure, the primary node 105 a attempts to inform thebackup through the monitor 108 via the heartbeat thread 107. When thebackup receives notification, it assumes the role of the primary node.Since the backup node has been processing the alternate routine 115concurrently, a result is available immediately for output.Subsequently, recovery time for this type of failure should be muchshorter than if both blocks were running on the same node. If theprimary node 105 a stops processing entirely, no update message will bepassed to the backup. The backup detects the crash by means of a localtimer in which timer expiry constitutes the time acceptance test.

The failed primary node transitions to a backup node, and by employing arecovery block reconfiguration strategy both nodes can be assured to notbe executing the same routine.

A distributed recovery block with real time process control is referredto as an extended distributed recovery block (EDRB) 102. The EDRBincludes a supervisor node 103 connected to the network to verifyfailure indications and arbitrate inconsistencies; and regular, periodicheartbeat status messages.

In EDRB, nodes responsible for control of the process and relatedsystems are called operational nodes and are critical. The operationalnodes perform real time control and store unrecoverable stateinformation. A set of dual redundant operational nodes is called a nodepair. Multiple redundant operation nodes are node sets.

Regular, period status messages are exchanged between node pairs andeach node pair in a node set. The messages are referred to asheartbeats. A node is capable of recovering from failures in itscompanion in standalone fashion, if the malfunction has been declared aspart of the heartbeat message. If a node detects the absence of itcompanion's heartbeat, it request confirmation of the failure from asecond kind of node called the supervisor. Although the supervisor isimportant to EDRB operation, the supervisor node 103 is typically notcrucial because its failure only impacts the ability of the system torecover from failures require its confirmation or arbitration. The EDRBsystem can continue to operate without a supervisor 103 if no otherfailures occur.

In FIG. 1 the software structure in a node pair is shown. Operationalnodes employ active redundancy. One node pair member is always active,the other is always standby if, it is functional. The active node 105 aexecutes a primary version of a control process in parallel with analternate version executed on the standby node. Both nodes check thecorrectness of the control outputs with the acceptance test 120.

Within an operational node, the EDRB is implemented as a set ofprocesses communicating between node pairs and the supervisor 103 tocontrol fault detection and recovery. The two processes responsible fornode-level fault decision making are the node manager 106 and themonitor 108. The node manager 106 determine the role of the local node(active or standby) and subsequently triggers the use of either theprimary 110 or the alternate routine 115. If the primary routineacceptance test is passed, the node manager 106 permits a control signalto be passed to the device drivers 130 under its control. If theacceptance test is not passed, the active node manager 106 a request thestandby node manager to promote itself to active and immediately sendout its result to minimize recovery time.

The monitor 108 associated with node manager 106 is concerned primarilywith generating the heartbeat and determining the state of the companionnode. The heartbeat is a ping or other rudimentary signal indicatingfunctionality of the respective node. When an operational node fails toissue a heartbeat, the monitor processes request permission from thesupervisor to assume control if not already in the active role. If thesupervisor 103 concurs that a heartbeat is absent, consent istransmitted and the standby node 105 b promotes itself to active node.

If the active node 105 a spuriously decides to become a standby node ora standby node makes an incorrect decision to assume control. As aresponse the supervisor node 103 will detect the problem form periodicstatus reports. It will then send an arbitration message to theoperation nodes in order to restore consistency.

In many computer networks, particularly in communication system, thesupervisory node 103 is critical, providing frame synchronization andconnection routes between the network and users. Thus, the loss of asupervisory node results in loss of the node function. Thus, there is aneed for a multiple redundant architecture in which not only are thenodes replicated, but also the network. In addition there is a need forimplementation of agent oriented software to facilitate thefunctionality of such an architecture.

SUMMARY OF THE INVENTION

An object of the invention is an improvement of a computer systemimplementing an extended distributed recovery block fault tolerancescheme comprising a supervisory node, an active node and a standby node.The active and standby node have a primary routine for executing asoftware function; an alternate routine for executing the softwarefunction; and an acceptance test routine for testing the output of theprimary routine and providing a control signal in response thereto. Theactive and standby nodes also having a device driver for receiving thecontrol signal, a monitor for communicating state information with oneor more active or standby nodes, and are operationally connected to anode manager for determining the operational configuration of the node.The primary routine is executed in response to a determination that thenode is in an active state and the alternate routine is executed inresponse to a determination that the node is in a standby state. Thesupervisory node coordinates the operation of the active node and thestandby node. The improvement being the primary and alternate routinesof one of the active or standby node are implemented with an applicationtask comprising a plurality of agent objects each operating as a finitestate machine operating in either a primary mode executing the primaryroutine or in an alternate mode executing the alternate routine.

Another object of the invention is an improvement of a computer systemimplementing an extended distributed recovery block fault tolerancescheme comprising a supervisory node, an active node, and a standbynode. The improvement being the primary and alternate routines of theactive and standby nodes are each implemented with a plurality ofdedicated application tasks each with a plurality of agent objectsoperating as a finite state machine in either a primary mode executingthe primary routine or in an alternate mode executing the alternateroutine. The determination of the mode of operation of the agents in aone of the plural dedicated application tasks is determinedindependently of the mode of operation of the agents in the other of theplural dedicated application tasks.

Still another object of the invention is an improvement of a computersystem implementing an extended distributed recovery block faulttolerance scheme having a supervisory node, an active node, and astandby node. The improvement being a primary and alternate routines ofthe active and standby nodes are each implemented with a plurality ofdedicated application tasks each with a plurality of agent objectsoperating as a finite state machine operating in either a primary modeexecuting the primary routine or in an alternate mode executing thealternate routine. Each of the agents is implemented with an attachmentlist comprising data common to the attachment list of at least one otheragent.

Yet another object of the invention is an improvement of a single bussoftware architecture for supporting hardware hot standby redundancywith a supervisor processing node. The improvement of adding a secondsupervisor processor node, alternatively in an active state, connectedto the bus to provide for a redundant supervisory node set.

Another object of the invention is an improvement of a communicationsystem with an active node and a standby node that form a node pair ornode set, each node with a node agent. The improvement of using areliable datalink between the heartbeat monitors of the node pair orset.

Another object of the invention is an improvement of a communicationsystem with an active node and a standby node that form a node pair ornode set, each node with a node agent. The improvement involvingsupporting automatic protection switching between multiple node sets orpairs using common agent architecture.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of an extended distributed recovery blockEDRB.

FIG. 2 is a representation of an exemplary state transition diagram forAgent Objects.

FIG. 3 is a common agent relationship diagram.

FIG. 4 is a representation of an exemplary node employing agentarchitecture.

FIG. 5 is a representation of an exemplary heartbeat message cell.

FIG. 6 is a representation of exemplary node sets employing a reliabledata link.

FIG. 7 is a representation of an exemplary dual redundancy protectionscheme employing a redundant supervisor node and a second data bus.

DETAILED DESCRIPTION

The implementation of EDRB of the present invention employs a hybridsolution, as it blends the agents with the structure and control of theEDRB. Application tasks are implement by agent objects that areinstances of C++ programming. The agent objects (agents) are implementedas finite state machines (circuit state machines) that recognized twodistinct modes of operation. One mode executes a primary routine block;the other executes an alternate routine block. An application taskperforms the acceptance test block and outputs the results for use bythe node manager in that processor node.

A circuit state machine 200, an example of which is shown in FIG. 2,implementing the agent objects is comprised of 5 states includingNot-Present 201, Restore 202, Stand-by 203, Active 204, and Out ofService 205. The circuit state machines are not limited to these states,more or less states are envisioned as required in certain applications.

The circuit state machines begin in the NOT PRESENT state 201 and staysin this state until a detected event is received. Once detected, theRESTORE State 202 is entered where the circuit is reset and circuitinitialization is performed. This transition can include successfuldiagnostic test execution as part of the initialization sequence. If aproblem arises during the transition, the state machine may betransitioned to the OUT OF SERVICE State 205 to await furtherinstructions. The OUT OF SERVICE State 205 is a holding state forsituations where fatal or unrecoverable errors have occurred. It is alsoa deliberate state to enter when conducting diagnostic test or whenattempting to restore normal operation.

The circuit state machine will stay in the RESTORE State 202 until aready event is received. A time can be provided to allow adequate timefor concurrent activity that may be required to initialize a circuit.Upon expiry the state machine automatically transition to the OUT OFSERVICE State 205.

Upon receipt of a ready event, the circuit state machine transition tothe STAND-BY state 203. In STAND-BY State 203 the circuit is identifiedas operation, but not in service for normal use. The circuit statemachine stays in the STAND BY state until an enable event is received,where it transition to the ACTIVE state 204. The ACTIVE State 204represents the state when the circuit is operational i.e. routingtraffic, monitoring defects, counting errors, and so on.

Software implements the circuit state machine state event matrix, eventprocedures and generic methods to provide a virtual behavior mechanism.The common Agent uses this generic behavior for software circuits, whereblocks of executable code 250 perform as though they are hot-swappablecomponents. In each of the state transitions depicted in FIG. 2, a chainof executable blocks of code are attached. These executable blocks ofcode are attachments which together an attachment list 251. When a statetransition occurs, each attachment in the Attachment List 251 for thattransition is executed in order. After each attachment runs, a statuscode is returned. If the status code is anything other than a success,execution of the chain is aborted.

Two additional execution chains are provided for handling the receipt ofmessages through he corresponding task service queue. One executionchain is provided for messages received when in the ACTIVE state, theother for messages received while in the STAND-BY state, these asdiscussed earlier are the primary and alternate routines, respectively.

When a message is received in the ACTIVE state, it is passed along toeach attachment in the primary execution chain 253 until the end isreached or a routine returns unsuccessfully. Likewise, when a message isreceived while in the STAND-BY state, it is passed along to eachattachment in the alternate execution chain (attachment list) 252. If amessage is received while the state machine is in any other state, it isignored. This supports the desired behavior where the agent object isonly operational when it is active or stand-by.

The common Agent object 300 relationship with neighboring externalentities is shown in FIG. 3. The Application Support Package (ASP)subsystem 301 (a operating system utility to provide state machines)acts upon the agent object by invoking its operation during taskinitialization and through the processing of state event registered atstartup. The Application Support Package (ASP) provides a generic tasklibrary that is used by application tasks. This library provides for astandardized internal task architecture and facilitates common softwaretest hooks. The generic task interfaces with the application'simplementation by means of specified user supplied hooks. The tasklibrary also uses all of the task level Runtime Library services thatthe application wishes to subscribe to. These services include FiniteState Machine dispatch, and a guaranteed FSM timer service.

As state machine event transitions occur and as service queue taskmessages are received, the common Agent objects 300 acts on theexecutable blocks of code 302 attached at startup or at any point afterstartup. The circuit state machine behavior can be directed by aredundancy node manager 303 task during condition when systemreconfiguration is required and resources in stand by become active forthose that have failed. The redundancy node manager can issue commandsto groups of agent objects instead of requiring software for eachexplicit function and procedure to invoke the reconfiguration process.

Common agent objects contain list of common Attachment objects which asdiscussed above are blocks of executable code. Agent objects may containsimilar or application-specific attachments added in such a fashion asto perform their intended roles and inherently support the redundantsystem architecture. The Attachment lists are dynamically modifiable.

FIG. 4 shows an example of an active or standby node 400, with aplurality of dedicated application tasks 404 a-c, implemented with aplurality of Agents 450. The application tasks implement the primary andalternate routine of the active or standby node, via the agents. Theagents 450 execute an attachment list for a primary 410 or alternate 415routine within the application task. All subject to the acceptance test.A plurality of agents within an application task may run the primary oralternate routines, which may or may not have different attachmentlists. The operation of the agents 450 in application task 404 a areindependent of the operation of the agents 450 in the other applicationtasks 404 b, 404 c etc.

A first set of Agents within an application task operate in the primarymode while the remainder of agents operate in the alternate mode. Theagents are configured such that a number of agents in a second setbackup a number of agents in the first set of agents. The number ofagents in each set may or may not be equal; furthermore each agent ofthe second set of agents may back up each of the agents in the firstset. Such a system allows for N- to M protection of the computer systemat the application task level.

During system initialization, agents register data ownership andsubscribe to data required for accomplishing assigned roles andprocesses. The data is common to all the agents. Blocks of the sameexecutable code that are shared by the Agents are contained in commonattachment lists. The Attachment lists are dynamically modifiable as afunction of the status of the computer system.

One or more agent objects can implement each of the application tasks.The application tasks perform the acceptance test block 420 and outputthe results for use by the node manager in that processor node. Theacceptance test block 420 is a test dedicated and contained within theapplication task. The node manager, upon acceptance sends the data tothe respective one or more device drivers 430.

Each node in the node pair or set is connected to it companion asdiscussed above via a heartbeat thread to the monitor and the node agentof each companion node. The heartbeat thread carries a heartbeat signal.The heartbeat contains the node roles, version and frame number, whichis incremented at the beginning of each new heartbeat frame. Preferablythe heartbeat thread is a reliable datalink between the monitors of thenode pair. For example, applying high-level data link control (HDLC)procedures would be a desirable implementation for the heartbeat thread,where the datalink message retransmission queues can be tuned to theneeds of the system in a deterministic fashion. Such an implementationis illustrated in the heartbeat message cell of FIG. 5.

The content of the heartbeat message cell 500 is shown in FIG. 5 inoctet format. The contents form either unnumbered 510, supervisory 520,or heartbeat information message 530 frames, depending on the state ofthe monitors in each participating node pair and an address frame 501.The message format enforces a level of integrity between node pairs tomanage standby sparing activation a signaling between field replacementunits (FRU). FRUs are units that service personnel can replace in thefield. The message is terminated with a frame check sequence (FCS) field540. Since this is a small message the FCS field 540 is an 8 bitsequence, invalid frames are frames which have fewer than 3 octets,contains a frame check sequence error, or contain an address that is notsupported.

The address field 501 consists of a command/response bit (C/R) 502, aservice access point identifier (SAPI) subfield 503 and a terminalendpoint identifier (TEI) subfield 504. The command/response bitidentifies a frame as either a command or a response. The backup nodesends commands with the C/R bit set to 0, and responses with the C/R bitset to 1. The primary node does the opposite, commands are sent with C/Rset to 1 and responses are sent with C/R set to 0. In conformance withHDLC high-level data link control procedures) rules, both node pairentities us the same datalink connection identifier composed of theSAPI-TEI pair. The SAPI is used to correspond the processor node slotwith the computer system connection. The TEI is used to map theconnection to a specific network interface.

The unnumbered (U) format 510 is used to provide data link controlfunction, primarily used in establishing and relinquishing link control.The Supervisory (S) format 520 is used to perform data link supervisorycontrol function such as acknowledge I frames, request transmissions ofI frames, and request temporary suspension of the transmission of Iframes. The function of N(R) and P/F are independent. Each supervisoryframe has an N(R) sequence number which may or may not acknowledgeadditional I-frames. The heartbeat information (I) format (I-frames) 530is used to perform normal information transfer between node pairs ornode sets regarding automatic protection switching and operationalstatus. The function of N(S), N(R) and P are independent. Each I-framehas an N(S) sequence number, an N(R) sequence number which may or maynot acknowledge additional I-frames, and a P bit that may be set to oor 1. K1 and K2 are signaling byte information maintained between nodepairs and sets of node pairs.

The poll/final bit is incorporated in all frames. The P/F bit serves afunction in both command frames and response frames. In command framesthe P/F bit is referred to as the P bit (poll) in response frames it isreferred to as the F bit (final) the P bit is set to 1 by a node pair tosolicit a response frame from the peer node. The F bit is set to 1 by anode pair to a response frame transmitted as a result of a solicitingcommand.

The receive sequence number N(R) is the expected send sequence number ofthe next received I frame. At the time that an I or S frame isdesignated for transmission, the value of N(R) is equal to the number ofI frames acknowledged by the node entity. N(R) indicates that the nodeentity transmitting the N(R) has correctly received all the I-framesnumbered up to and including N(R)−1. The send sequence number N(S) isthe send sequence number of transmitted I frames. It is only used inI-frames. At the time that an in-sequence I frame is designated fortransmission, the value of N(S) is set equal to the current sequencenumber for the I frame to be transmitted.

The supervisory command sequence comprises receive ready, reject andreceive not ready commands. The unnumbered control function includeexpand mode, disconnected mode, disconnect unnumbered acknowledgment andframe reject.

The EXPAND mode command is used to place the addressed backup or primarynode into multiple frame acknowledged operation. A node pair confirmsacceptance of any expand mode command by the transmission at the firstopportunity of an unnumbered acknowledgement response. Upon acceptanceof this command, the node pair entity sequence and transmission counterare set to 0. The transmission of an expand mode command indicates theclearance of all exception conditions. Exception conditions are delays,retransmit counters, erred messages of other condition outside of normalmessages. Previously transmitted I frames that are unacknowledged whenthe EXPAND mode command is processed remain unacknowledged and arediscarded.

The Disconnect command terminates the multiple frame operation, such aswhen the network operator decides to take a node pair out of service orchange the backup node. The node pair entity receiving the disconnectcommand confirms the acceptance by the transmission of an unnumberedacknowledgement response. The node pair entity sending the disconnectcommand terminates the multiple frame operation upon receipt of theunnumbered acknowledgment response or the disconnected mode response.

The Receive ready command indicates when a node set is ready to receivean I-frame, acknowledge previously received I frames or clear a busycondition indicated by an earlier transmission of a receive not readycommand by the same node set. The Reject command is used by a node pairentity to request retransmission of I frames starting with the framenumbered N(R). The value of N(R) in the reject frame acknowledges Iframes number up to and including N(R)−1. Only one rejection exceptioncondition for a given direction of information transfer is establishedat a time. The Rejection condition is cleared upon the receipt of an Iframe with an N(S) equal to the N(R) of the reject frame.

The receive not ready command indicates a busy condition: that is, atemporary inability to accept additional incoming I frames. The valueN(R) in the receive not ready command acknowledges I-frames numbered upto and including N(R)−1. The unnumbered response acknowledges thereceipt and acceptance of mode setting commands EXPAND and DISCONNECTED.The disconnected mode response reports to its peer that the heartbeatlink is in a state such that multiple frame operation cannot beperformed. The Frame Reject response reports an error condition notrecoverable by retransmission of the identical frame.

A configuration of nodes employing a reliable data link is shown in FIG.6. Node pair 601 includes processor nodes 605 a and 605 b are connectedby a reliable data link 680. Similarly, node set 602 includes multipleprocessor nodes 605 c-e which are connected by reliable data links 680.Each node contains a node agent 650 and a monitor. Again the node agent650 is preferable an instance of C++ programming and resides on thenode. The device or line interfaces relay data messages to the nodeagent 650, which can include externally generated automatic protectionswitching commands and line interface status. The node pairs or nodesets also may include a recovery agent.

The node agent though the monitor accepts and filter line interfacestatuses and external automatic protection switching commands though thereliable data link and provide a more sophisticated communicationbetween node agents in a node pair or set. As a result if a card failureoccurs, i.e. the node goes down, the reliable data link will break, andthus like discussed earlier, the standby node will attempt to go online, unless preempted by the supervisor node or the recovery agent.However, in case of a line failure, the data link stays up and theactive processor node signals standby processor node of failure and thestandby node becomes active unless preempted.

FIG. 7 shows an exemplary node pair 701 and node set 702 with dualredundant supervisor nodes. Each node 705 a-e containing a monitor 708attached each other in the node set via a heartbeat thread 707. Thenodes execute application tasks 704 implemented by agents 750, which runa primary 710 or alternate 715 routine. For each node set, one node isactive while the remainders are in standby mode. The first supervisornode 730 is active and connected to the node sets via a first bus 732.The second supervisor node 731 is connected to node sets via a secondbus 733. The first and the second buses are operationally connected tothe processor nodes. The supervisor nodes abstractly operate much like anode pair, in that when one is active the other is in standby mode.Likewise the supervisors nodes 730 and 731 may employ the use of aheartbeat signal between their respective monitors 708. Each of thesupervisor nodes is connected to the processor nodes via a different busor buses. A multitude of addition supervisor nodes may be used, alongwith additional buses configure much in the same manner as describedabove. Implementation of more than two supervisor nodes allows formultiple redundancy in which multiple stand by supervisor nodes backupmultiple active supervisor nodes.

A plurality of agents may reside upon the supervisor nodes including, aspreviously discussed a recovery agent, which is an instance of C++programming. The recovery agent directs or overrides the transition ofnodes between active and standby. The recovery agent fulfils one or moreof the supervisory roles.

The processor and agent architecture described herein is particularlysuited for use in a point-to-multi-point wireless communication systemused to communicate from a central location to each of a plurality ofremotes sites where reliable connections are required. Such a systemthat provides high speed bridging of a physical gap between a pluralityof processor based systems, is ultimately dependent on the faulttolerance and recovery capability of the processors which comprise itsstructure.

Although the invention has been described in a preferred form with acertain degree of particularity, it is understood that the presentdisclosure of the preferred form has been made only by way of example,and that numerous changes in the details of construction and combinationand arrangement of parts may be made without departing from the spiritand scope of the invention as hereinafter claimed. It is intended thatthe patent shall cover by suitable expression in the appended claims,those features of patentable novelty that exists in the inventiondisclosed.

1-47. (canceled)
 48. A single bus software architecture for supportinghardware hot standby redundancy with a supervisor processing node, theimprovement of adding a second supervisor processor node, alternativelyin an active state, connected to said bus to thereby provide for aredundant supervisory node set.
 49. The software architecture of claim48 wherein when one of said supervisor processor nodes is in an activemode, the other supervisor processor node is in a stand-by mode.
 50. Thesoftware architecture of claim 49 further comprising one or moreadditional busses, wherein said active supervisor processor node isattached to at least one bus and wherein said stand-by supervisorprocessor node is attached to at least one bus.
 51. The softwarearchitecture of claim 50 wherein said active supervisor processor nodeand said standby supervisor processor node are attached to differentbusses.
 52. The software architecture of claim 48 wherein said softwarearchitecture comprises a computer system.
 53. The software architectureof claim 52 wherein said computer system comprises a communicationsystem.
 54. The software architecture of claim 53 wherein saidcommunication system is a point-to-multipoint communication system. 55.The software architecture of claim 54 wherein said communication systemoperates in the millimeter wave frequency spectrum.
 56. The computersystem of claim 55 wherein said communications system is a time divisionmultiple access system for communicating data in a frame format.
 57. Thecomputer system of claim 56 wherein the data density within each frameis dynamically variable.
 58. The computer system of claim 56 whereinsaid communications system is a time-division duplex system.
 59. Thecomputer system of claim 58 wherein said communications system is anadaptive time division duplex system.
 60. The computer system of claim59 wherein the forward/reverse ration is dynamically configurable.